With modern filesystems, a large amount of state must be held or cached in memory in order to provide good performance and implement sophisticated storage management features. Quickly recovering and loading this state back into memory is important when the filesystem starts up, recovers from a crash, and for certain types of failover operations. As filesystems become larger and more complex, the amount of in-memory state increases and the amount of on-disk state that may need to be scanned to recover the in-memory state becomes larger. Very quickly, scanning large amounts of on-disk state in order to discover and recover the needed in-memory state may become unacceptably slow.
In some systems, most if not all of the metadata in a hybrid filesystem is stored in a particular type of persistent, underlying storage. For example, the metadata in the hybrid filesystem can be stored on solid-state disk (SSD) drives. Specifically, for example, the SSD storage comprises flash storage. The total flash space may be different on each model and generally increases each year as new larger and higher performance models are introduced. Conventionally, metadata state data related to the metadata is stored in a memory or some other type of non-persistent (e.g., temporary) storage. However, when the system crashes or is rebooted, the metadata state data in the memory is lost.
Upon restart or reboot of the system, the metadata state data associated with the system's metadata should be rebuilt in the memory. Conventionally, in some systems, the amount of (e.g., flash) storage to scan to discover the needed metadata was proportional to the size of the flash storage. Put another way, a substantial portion of the flash storage is typically scanned to discover the needed metadata. The metadata is discovered so that the metadata state data related to the metadata can be rebuilt in memory. However, with the increasing size of flash storage on a system, the total time that is needed to scan the flash storage for discovering metadata objects may become too long and is in danger of exceeding a predetermined upper bound limit on the time to perform startup.
Furthermore, in some systems, a log-structured filesystem is used. In a log-structured filesystem, objects are written to the filesystem in an append-only fashion, meaning that old versions of an object are not modified (but rather persist until they are garbage collected) and that new versions of the object are created at different locations in storage. As a consequence of writing objects in the append-only fashion, a scan for metadata objects may discover multiple versions of the same object in storage. Also, log-structured filesystems usually need a garbage collection process to reclaim space from dead objects in the filesystem. A garbage collection process in a log-structured filesystem may move a version of an object by making a new copy of that version of the object in a different location in storage without immediately reclaiming the earlier copy of the version of the object. As a result, a scan for metadata objects may also discover multiple copies of the same version of the object in the filesystem.
During the filesystem startup, the only version of an object that is relevant to the startup is the most recent one. Thus, effort and resources are spent on identifying a copy of the most recent version of an object among multiple copies and/or multiple versions of the object that may be discovered in the flash storage.